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Abstract 

GAIM-GM  is  an  operational  Kalman  filter  data  assimilation  model  of  the 
ionosphere  that  can  assimilate  data  from  GPS  total  electron  content  (TEC),  ionosonde 
electron  density  profiles,  and  satellite -based  in-situ  electron  densities.  The  Air  Force 
Weather  Agency  (AFWA)  uses  GAIM-GM  to  specify  and  forecast  the  ionosphere, 
however,  an  in-depth  investigation  into  the  accuracy  of  these  forecasts  has  not  been 
completed.  GAIM-GM  output  obtained  from  four  cases  run  from  combinations  of 
geomagnetic  and  solar  activity  was  used  to  detennine  GAIM-GM  forecast  accuracy. 
Forecast  accuracy  was  detennined  through  the  use  of  a  skill  score  as  well  as  other 
statistical  tools  to  include  root  mean  square  error  (RMSE)  and  percent  differences 
between  the  forecasts  and  the  observations.  It  is  determined  that  overall,  GAIM-GM 
forecast  output  is  more  representative  than  the  climatological  reference  90%  of  the  time 
indicating  that  GAIM-GM  can  therefore  forecast  short  term  fluctuations  in  the 
ionosphere.  Skill  scores  decrease  at  night  and  GAIM-GM  shows  an  overall  diurnal  bias 
to  under  forecast  at  night  and  over  forecast  during  the  day.  The  RMSE  increases  during 
daytime,  geomagnetic  stonning  and  during  solar  maximum. 
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VERIFICATION  OF  GLOBAL  ASSIMILATION  OF  IONOSPHERIC 


MEASUREMENTS  GAUSS  MARKOV  (GAIM-GM)  MODEL  FORECAST 

ACCURACY 


I.  Introduction 


1.1  Motivation  and  Background 

Since  the  dawn  of  the  space  age,  technological  advances  such  as  long  range 
communication,  Global  Positioning  System  (GPS)  navigation  and  space  based 
surveillance  have  advanced  at  a  torrid  pace  and  continue  to  do  so.  These  space  based 
capabilities  are  routinely  used  and  exploited  in  both  military  and  civilian  operations.  Our 
collective  dependence  on  these  capabilities  is  unquestioned,  thus,  it  is  imperative  to  fully 
understand  how  the  space  environment  behaves,  since  it  impacts  these  systems. 

The  ionosphere  is  the  near  earth  subset  of  the  space  environment  that  is  located 
roughly  between  90  km- 1500  km  above  the  earth.  Satellite,  navigation  and 
communication  based  capabilities  operate  in  this  dynamic  environment.  It  is  critical, 
therefore,  to  fully  understand  and  accurately  model  the  ionosphere  in  order  to  mitigate 
potential  threats  to  these  space  based  capabilities.  These  threats  include,  but  are  not 
limited  to,  radio  communication  blackouts  and  navigational  errors  associated  with 
fluctuations  in  the  ionosphere. 
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Ionospheric  and  space  weather  models  are  improving,  owing  in  part,  to  the 
increase  in  available  ionospheric  data  from  GPS  satellites,  military  satellites  and  land 
based  ionospheric  readings.  The  improvement  in  space  weather  and  ionospheric  models 
gives  military  and  civilian  decision  makers  greater  tools  and  confidence  in  predicting  the 
behavior  of  the  ionosphere  and  therefore  greater  ability  to  plan  and  make  decisions 
critical  to  military  operations  and  national  security. 

The  Global  Assimilation  of  Ionospheric  Measurements  Gauss-Markov  model 
developed  at  Utah  State  University  and  currently  operational  at  the  Air  Force  Weather 
Agency  is  a  physics  based  data  assimilation  model  of  the  ionosphere  that  generates  both 
specifications  (nowcasts)  and  forecasts  of  the  ionosphere.  Validation  studies  have  been 
conducted  to  detennine  the  accuracy  of  the  specifications,  but  little  work  has  been 
focused  on  verifying  forecast  accuracy. 

1.2  Research  objectives 

This  work  aims  to  detennine  the  forecast  accuracy  of  GAIM-GM  as  run  by 
AFWA.  The  end  goal  is  to  determine  the  accuracy,  strengths  and  weaknesses  of  the 
model.  This  information  will  aid  the  developers,  operators  and  end  users  of  GAIM-GM 
to  improve  upon  and  better  understand  the  limitations  of  the  model.  This  is  done  by 
running  GAIM-GM  for  several  cases  that  include  variations  in  geomagnetic  and  solar 
activity  as  well  as  focusing  over  both  the  entire  globe  and  CONUS.  The  forecasts  and  a 
climatological  reference  background  are  compared  to  the  ionospheric  specification  to 
detennine  the  forecast  accuracy  with  respect  to  climatology  via  a  forecast  skill  score. 
Forecast  accuracy  is  evaluated  through  statistical  tools  to  include  root  mean  square  enor 
(RMSE)  and  percent  differences  between  the  forecasts  and  the  specifications. 
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1.3  Document  Structure 


There  are  5  chapters  in  this  work.  Chapter  I  is  an  introduction.  Chapter  II 
provides  background  infonnation  about  GAIM-GM,  the  Ionospheric  Forecast  Model 
(IFM),  the  data  used  in  the  research,  the  Kalman  filter,  and  the  statistical  tools  used  to 
detennine  forecast  accuracy  to  include  the  forecast  skill  score.  Chapter  III  focuses  on  the 
methodology  in  executing  the  forecast  accuracy  verification.  Such  items  include,  the 
geophysical  conditions  of  each  case,  how  GAIM-GM  is  run,  what  data  is  used,  how  the 
skill  scores  are  determined  and  pertinent  assumptions  to  ensure  the  model  is  operated 
similarly  to  how  it  is  operated  at  AFWA.  Chapter  IV  contains  the  results  and  analysis  of 
the  verification  findings  and  Chapter  V  provides  a  summary  of  the  work  as  well  as 
conclusions  and  suggestions  for  continuing  work  on  this  topic. 
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II.  Background 


This  chapter  lays  the  groundwork  for  the  remainder  of  the  paper.  It  begins  by 
describing  the  ionosphere  and  then  moves  on  to  introducing  GAIM-GM  and  its  input 
model,  the  IFM.  Sections  on  the  data  ingested  into  GAIM-GM,  the  Kalman  filter  and 
skill  scores  are  included.  Finally,  previous  validation  research  is  summarized  to  justify 
the  assumptions  for  the  methodology  used  in  performing  the  forecast  accuracy 
verification. 

2.1  The  Ionosphere  and  Plasmasphere 

The  ionosphere  is  located  from  60  km  to  more  than  1000  km  above  Earth  and  is 
so  named  due  to  the  presence  of  ions  within  this  region.  Neutral  molecules  are  ionized 
from  extreme  ultraviolet  (EUV)  and  Soft  X-ray  radiation  from  the  sun  in  a  process  called 
photoionization.  These  ions  are  then  subject  to  the  processes  of  chemical  reactions, 
diffusion,  recombination,  and  transport  by  the  neutral  winds.  Variations  in  latitude,  the 
day-night  sector,  the  Earth’s  magnetic  field  and  the  previously  mentioned  processes 
create  various  regions  within  the  ionosphere  denoted  the  D,  E,  F  regions  and  the  topside 
ionosphere,  illustrated  in  Figure  1  ( Schunk  and  Nagy,  2009). 

The  D  region  is  found  between  60  km- 100  km  and  is  heavily  influenced  by 
complex  chemical  reactions.  The  E  region  is  located  between  100  km- 150  km  and  is 
likewise  dominated  by  chemical  processes,  albeit  less  than  the  D  region,  and  is 
characterized  as  a  weakly  ionized  plasma  ( Schunk  and  Nagy,  2009). 

The  partially  ionized  F  region  is  broken  up  into  two  components,  the  Fi  and  F2 
layers  located  between  150  km-250  km  and  around  300  km  respectively.  Ion  transport 
processes  begin  to  dominate  over  chemical  processes  in  this  region  and  the  point  where 
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chemical  and  transport  processes  balance  is  denoted  the  F2  peak  as  ion  densities  reach  a 
maximum  at  this  juncture.  The  region  above  the  F2  peak  is  denoted  the  topside 
ionosphere,  characterized  by  fully  ionized  plasma  (Schunk  and  Nagy,  2009). 

The  protonosphere  or  plasmasphere  is  the  doughnut  shaped  region  around  the 
Earth,  collocated  and  above  the  topside  ionosphere  characterized  by  high  concentrations 
of  H+  ions  ( Schunk  and  Nagy,  2009).  The  altitude  range  of  GAIM-GM  encompasses  part 
of  the  plasmasphere. 
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Figure  1.  Ionospheric  profile  for  selected  ions  and  electrons  as  well  as  the  layers  of  the  ionosphere. 
(Adapted  from  Schunk  and  Nagy,  2009) 
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2.2  GAIM-GM 


The  GAIM-GM  model  is  a  time  dependent  physics  based  model  that  uses  the 
Ionospheric  Forecast  Model  (IFM)  as  a  background  model  upon  which  perturbations  are 
imposed  via  a  Kalman  filter  data  assimilation  scheme.  GAIM-GM  can  use  either  real 
time  or  historical  data  to  generate  ionospheric  specifications  as  well  as  forecasts  up  to  24 
hours  out  from  the  current  specification  ( Scherliess  et  al.,  2006). 

GAIM-GM  outputs  three  dimensional  ion  and  electron  densities  of  the  ionosphere 
at  altitudes  from  90  km  to  1400  km  at  15  minute  time  intervals  as  well  as  total  electron 
content  (TEC),  peak  density  of  the  F2  region  (NiiiFj)  and  height  of  the  F2  peak  ( h,„F2 ). 
TEC  is  the  total  number  of  electrons  between  two  points  given  in  units  of  electrons/nr, 
for  example,  between  a  satellite  and  a  ground  based  GPS  receiver,  and  is  sometimes 
reported  in  TEC  units  (TECU),  with  one  TECU  equivalent  to  1016  electrons/m2.  GAIM- 
GM  global  resolution  is  4.67  degrees  latitude  by  15  degrees  longitude.  Its  vertical 
resolution  is  4  km  in  the  E  region  and  20  km  in  the  F  region,  the  same  vertical  output 
resolution  as  the  IFM.  GAIM-GM  can  also  be  run  in  a  regional  mode  with  a  finer 
resolution  ( Scherliess  et  al.,  2006). 

GAIM-GM  is  able  to  assimilate  many  data  types  and  is  modular  such  that  new 
data  types  will  be  able  to  be  incorporated  into  the  data  assimilation  scheme.  Data  that 
GAIM-GM  can  assimilate  include  in  situ  electron  densities  from  governmental  satellites, 
namely  the  Defense  Meteorological  Satellite  Program  (DMSP)  and  National  Oceanic  and 
Atmospheric  Administration  (NOAA)  satellites,  bottomside  electron  density  profiles,  and 
line  of  sight  TEC  measurements  from  thousands  of  ground  stations  and  GPS  satellites 
among  others  ( Schunk  et  al.,  2004).  The  highlighted  data  types  correspond  to  those  that 
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are  ingested  in  the  operational  model  currently  in  use  at  AFWA.  Other  data  that  GAIM- 
GM  can  ingest  but  that  are  not  currently  used  by  AFWA  include  data  from  UY  sensors 
aboard  the  DMSP  satellites,  measurements  from  various  satellites  to  include  the  TOPEX 
satellite  used  in  previous  model  verification  studies,  and  COSMIC  occultation  data, 
among  others.  Figure  2  shows  the  data  platforms  that  GAIM-GM  can  assimilate  along 
with  the  steps  involved  in  operating  GAIM-GM  and  the  relationships  between  the  IFM, 
data,  Kalman  filter  and  outputs.  The  first  step  in  GAIM-GM  is  to  set  forth  the  time  and 
geophysical  conditions  of  the  model  run.  Next  the  IFM,  or  physics  based  model,  is  run  to 
generate  a  background  ionosphere.  From  here,  data  is  ingested  and  assimilated.  The 
background  ionosphere,  in  conjunction  with  the  assimilated  data  produces  the  current 
state  ionospheric  specification  as  well  as  forecasts  from  this  specification  out  to  24  hours. 


STEP  0 


Figure  2.  Flow  chart  of  GAIM-GM  operational  sequence.  Shown  are  the  steps  involved  in  operating 
GAIM-GM,  data  that  can  be  assimilated,  input  parameters  and  outputs  (Adapted  from  Schunk, 
2004). 
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2.2.1  Ionospheric  Forecast  Model  (IMF) 

The  Ionospheric  Forecast  Model  is  a  physics-based  numerical  model  of  the  global 
ionosphere.  The  IFM  calculates  time-dependent  density  distributions  in  three  dimensions 
for  the  following  ions:  NO+,  C>2+,  N 2  and  0+,  in  the  E  region.  It  also  calculates  density 
distributions  for  the  major  ions,  0+  and  NO+  as  well  as  for  minor  ions  N2+  and  CF+,  in  the 
F  region.  Electron  and  ion  temperatures  are  also  calculated  in  the  E  and  F  regions. 
Furthennore,  H+  densities  are  calculated  in  the  F  region  and  topside  ionosphere  (Schunk 
etal.,  1997). 

The  IFM  encompasses  the  region  90  km- 1400  km  above  the  earth  with  a  spatial 
resolution  of  4  km  in  the  E  region  and  20  km  in  the  F  region.  Using  either  geographic  or 
geomagnetic  coordinate  systems,  the  IFM  generates  density  and  temperature  outputs  at  a 
resolution  of  3  degrees  latitude  by  7.5  degrees  longitude  ( Schunk  etal.,  2004). 

The  IFM  needs  only  several  basic  input  drivers  to  run.  These  include  F10.7  solar 
flux  proxy  number  (indicates  the  level  of  solar  activity),  year,  day,  start  time,  duration 
and  3  hourly  Kp  values  (indicates  level  of  geomagnetic  storming)  from  3  hours  prior  to 
the  start  time  through  the  end  of  the  model  run  ( Schunk  et  al.,  2004).  Due  to  its 
dependence  on  these  simple  drivers  as  well  as  empirical  models  to  derive  the  output,  the 
IFM  is  more  representative  of  the  ionospheric  climatology  than  it  is  for  shorter  term 
weather  variations  ( Sojka  et  al.,  2007). 

The  output  obtained  from  IFM  is  the  result  of  the  numerical  solutions  to  the 
energy,  momentum,  and  ion  and  electron  continuity  equations.  The  physical  processes 
taken  into  account  in  this  model  are  the  following:  (1)  field  aligned  diffusion  due  to 
density  and  temperature  gradients,  gravity  and  the  ambipolar  electric  field,  (2)  cross  field 
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electrodynamic  drifts  due  to  the  magnetopsheric  and  dynamo  electric  fields,  (3)  ion 
production  from  UV  and  solar  radiation,  resonantly  scattered  solar  radiation,  starlight, 
and  auroral  electron  precipitation,  (4)  numerous  energy  dependent  chemical  reactions,  (5) 
thennospheric  winds,  (6)  neutral  composition  changes,  (7)  thermal  conduction,  (8)  many 
elastic  and  inelastic  heating  and  cooling  processes.  Also,  the  geomagnetic  and 
geographic  polar  offset  is  accounted  for  in  the  IFM  ( Schunk  et  al.,  2004). 

2.2.2  Data 

GAIM-GM  can  assimilate  a  wide  variety  of  data  types.  The  operational  model  run 
at  AFWA  currently  ingests  three  types:  GPS  slant  TEC,  bottomside  electron  density 
profiles  obtained  from  ionosondes,  and  in  situ  electron  density  readings  from  DMSP 
satellites  from  the  Special  Sensors-Ions,  Electron,  and  Scintillation  (SSIES)  instrument. 
The  quantity  of  data  ingested  can  also  be  controlled  by  editing  the  lists  of  GPS  ground 
sites  and  ionosonde  sites  that  GAIM-GM  is  programmed  to  accept  as  well  as  specifying 
the  DMSP  satellites  from  which  to  accept  data. 

2.2.2. 1  GPS 

Data  obtained  from  GPS  is  extremely  valuable  due  to  the  spatial  and  temporal 
coverage  that  it  provides.  GPS  consists  of  both  satellite  and  ground  based  stations  that 
communicate  through  the  ionosphere.  In  obtaining  proper  time  and  location 
measurements,  GPS  must  take  into  consideration  the  ionospheric  environment  and  does 
so  by  computing  the  electron  content  in  the  path  between  satellites  and  ground  based 
receiving  stations.  This  is  accomplished  by  detecting  the  time  delay  from  two  radio 
signals  of  different  frequency  between  the  satellite  and  the  ground  receiver  and  relating 
this  to  electron  density  ( Ondoh ,  2001).  These  electron  readings  are  called  slant  TEC  and 
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can  be  obtained  and  assimilated  in  near  real-time.  They  are  also  archived  by  NOAA  and 
made  available  from  their  Continuously  Operating  Reference  Station  (CORS)  website. 
Ground  based  GPS  receivers  may  be  in  contact  with  more  than  one  satellite  at  a  time  and 
may  therefore  provide  more  than  one  slant  TEC  measurement  at  a  time. 

22.2.2  Ionosonde 

An  ionosonde  is  a  type  of  radar  that  transmits  radio  waves  of  different  frequencies 
into  the  ionosphere.  The  waves  are  reflected  at  the  point  where  the  radio  frequency 
transmitted  equals  the  plasma  frequency  of  the  ionosphere.  The  radio  waves  then  travel 
back  and  are  picked  up  by  the  receiver.  From  the  time  difference  between  the 
transmission  and  reception  of  the  radio  wave,  a  height  is  obtained.  If  the  propagation 
speed  is  assumed  to  be  the  speed  of  light,  then  the  height  is  virtual.  To  obtain  the  actual 
height  of  reflection,  the  group  velocity  of  the  propagating  wave  must  be  determined.  The 
electron  density  at  this  height  is  detennined  via  the  plasma  frequency  equation  given  by 


e  2  ne 


M 


4n2me  s0 


(i) 


Where  f  is  the  frequency,  e  is  the  electron  charge,  ne  is  the  electron  density,  me  is 
the  mass  of  an  electron  and  s0  is  pennittivity  ( Ondoh ,  2001).  A  bottomside  electron 
density  profile  of  the  ionosphere  is  obtained  from  the  ionosonde  and  GAIM-GM 
assimilates  this  data. 
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2.2.23  DMSP 


The  Defense  Meteorological  Satellite  Program  satellites  are  part  of  a  government 
run  satellite  program  that  observe  the  earth,  oceans  and  study  their  physical  processes. 
These  satellites  are  sun  synchronous  polar  orbiting  satellites  that  fly  at  an  altitude  of  -830 
km  and  are  equipped  with  several  types  of  sensors.  The  Special  Sensors-Ions,  Electron, 
and  Scintillation  (SSIES)  instrumentation  aboard  these  satellites  collect  in  situ  ion  and 
electron  density  readings  that  are  input  into  GAIM-GM.  This  is  done  with  a  suite  of 
sensors  located  on  the  satellite  that  include  a  retarding  potential  analyzer,  ion  drift  meter, 
a  scintillation  meter,  and  a  Langmuir  probe.  These  sensors  work  together  to  determine 
the  flow,  temperature,  angle  and  current  of  the  plasma  in  the  ambient  satellite 
environment.  From  these  sensors  the  electron  density  is  measured  and  this  data  is  input 
into  GAIM-GM  ( Delorey  et  ai,  1989). 

2.2.3  Kalman  Filter 

Data  quality  and  availability  for  the  space  environment  continue  to  increase, 
making  space  weather  data  assimilative  models  more  reliable.  In  order  to  leverage  the 
data  to  improve  the  model  accuracy  of  GAIM-GM,  a  Kalman  filter  data  assimilation 
scheme  is  used.  The  data  ingested  into  GAIM-GM  is  run  through  the  Kalman  filter, 
adding  ionospheric  perturbations  to  the  IFM  to  generate  more  accurate  specifications  and 
forecasts  ( Scherliess  et  al.,  2006). 

The  Kalman  filter  is  a  sequential  least  squares  (best  fit)  procedure  that  is  expert  in 
taking  in  various  data  types  into  the  model  and  properly  assigning  confidence  to  each 
data  assimilated.  The  data  is  used  to  further  refine  the  specification  from  which  the 
forecast  is  generated  {Howe,  1998).  When  inputting  data  into  the  model,  it  is  necessary 
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to  associate  errors  with  the  ingested  data.  For  GAIM-GM,  a  statistical  Gauss-Markov 
model  from  a  background  error  covariance  matrix  is  used.  This  matrix  was  derived  from 
1107  separate  2  day  runs  of  IFM  taken  from  varying  ionospheric  drivers  and  geomagnetic 
conditions  ( Thompson  et  al.,  2006). 

2.3  Skill  Score 

A  common  measure  in  detennining  the  accuracy  of  any  forecast  model  is  the  use 
of  a  model  skill  score.  Three  components  are  needed  to  generate  a  single  skill  score. 
These  typically  take  the  form  of  a  forecast,  an  observation  and  a  reference.  The  idea  is  to 
compare  both  the  forecast  and  a  reference  (typically  climatology)  to  an  observation  to 
detennine  which  is  more  representative,  in  this  case  with  respect  to  the  ionosphere.  This 
forecast  skill  is  computed  by  comparing  the  root  mean  square  error  (RMSE)  between  the 
forecast  and  the  observation  to  the  RMSE  between  the  reference  and  the  observation. 
The  first  RMSE  compares  the  error  between  the  forecast  and  the  observation  through  the 
following  equation 


M 

1  V1 

RMSE  = 

M  Aj  ~~  °m^ 

A 

m=l 

(2) 


Where  F  denotes  the  forecast  and  O  denotes  the  observation.  The  second  compares  the 
error  between  the  reference  and  the  observation  as  given  by 
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(3) 


Where  R  is  the  reference  and  O  is  the  observation.  The  skill  score  is  then  derived  from 
these  two  errors  by  the  equation 


Skill  Score  =  1  — 


RMSE  ^ 
RMSEref/ 


(4) 


Skill  scores  are  therefore  bounded  by  1  for  a  perfect  forecast  as  this  indicates 
perfect  agreement  between  the  forecast  and  the  observation.  Skill  scores  approach  an 
undefined  value  or  get  largely  negative  when  the  reference  is  more  representative  than 
the  forecast.  Therefore,  skill  scores  greater  than  zero  imply  that  the  forecast  was  more 
representative  than  the  reference  when  compared  to  the  observation.  A  zero  skill  score 
indicates  that  the  model  forecast  and  the  reference  were  equivalent  in  their  representation, 
and  a  negative  score  states  that  the  reference  was  more  representative  than  the  forecast 
output  (Wilks,  2006).  The  choice  for  the  reference  is  important  as  models  and  references 
have  inherent  strengths  and  weaknesses  that  must  be  taken  into  consideration  when 
computing  skill  scores.  In  other  words,  some  models  favor  climatology  while  others 
focus  in  on  more  short  term  weather  phenomena. 
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2.4  Previous  Validations 


This  section  outlines  some  of  the  notable  previous  work  with  regard  to  GAIM- 
GM  specifications  and  forecast  validation.  Also  included  is  a  summary  of  work  done  by 
Thompson  et  al.  (2006)  regarding  model  specification  accuracy  given  various  data  inputs. 
The  objectives  are  to  outline  what  has  been  done  previously  to  vet  the  output  quality  of 
the  model,  to  detennine  GAIM-GM  specification  accuracy  and  use  that  as  a  benchmark 
for  comparisons  in  this  work,  and  to  illuminate  the  need  to  properly  verify  GAIM-GM 
model  forecast  accuracy. 

2.4.1  Specification  Validation 

Various  validation  studies  to  determine  the  accuracy  of  the  specification  of 
GAIM-GM  have  been  carried  out  by  both  AFRL  and  USU.  The  team  at  AFRL  tested 
operationally  significant  parameters  that  include  the  maximum  usable  frequency  (MUF) 
for  HF  communications.  Because  it  is  impractical  and  very  difficult  if  not  impossible  to 
ubiquitously  measure  the  entire  ionosphere,  the  validation  and  verification  studies  of 
GAIM-GM  have  typically  focused  on  point  locations  or  been  compared  to 
climatologically  based  empirical  models.  For  example,  AFRL  validated  the  MUF  over 
Australian  ionosonde  sites  and  compared  GAIM-GM  output  with  these  ionosonde 
measurements  as  well  as  to  climatologically  based  monthly  median  MUF.  They  found 
that  GAIM-GM  errors  for  monthly  median  MUF  values  were  10%  for  daytime  and  16% 
at  night  ( McNamara  et  al.,  2008). 

Utah  State  University,  likewise,  has  conducted  validation  studies  of  GAIM-GM. 
Zhu  et  al.  (2006)  published  a  comprehensive  validation  study  comparing  GAIM-GM 
output  to  TEC  measurements  obtained  from  the  TOPEX  satellite.  The  TOPEX  satellite 
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was  launched  on  August  10,  1992,  and  its  main  mission  is  to  detennine  sea  surface 
heights.  Because  TOPEX  is  a  nearly  sun  synchronous  satellite  that  orbits  the  earth  at 
-1350  km,  it  is  well  placed  to  detennine  TEC  for  the  same  range  of  the  ionosphere  as 
IFM.  The  look  path  from  TOPEX  to  the  Earth  encompasses  much  of  the  ionosphere 
making  it  suitable  as  an  observational  tool.  The  satellite  is  equipped  with  two  dual¬ 
frequency  altimeters  and  from  this  instrumentation  TEC  estimates  are  obtained.  The 
TEC  comparisons  were  made  between  TOPEX  and  IFM,  the  physics  based  background 
model  for  GAIM-GM.  It  should  be  noted  that  the  validation  in  this  case  is  not  explicitly 
for  GAIM-GM  but  rather  the  GAIM-GM  background.  The  results  indicate  that  the  IFM 
and  TOPEX  TEC  measurements  are  consistent,  with  daytime  readings  outperfonning 
nighttime.  Quantitatively,  the  TEC  differences  are  typically  below  20%  with  some  values 
within  10%  (Zhu  et  al.,  2006).  The  80%  -  90%  agreement  between  the  TOPEX  TEC 
measurements  and  IFM  is  the  basis  for  using  GAIM-GM  specifications  as  the  observation 
for  the  skill  score  comparisons. 

2.4.2  Data  Impacts 

Since  GAIM-GM  is  able  to  assimilate  various  types  and  quantities  of  data,  it  is 
important  to  detennine  how  changes  in  data  type  and  quantity  alter  the  output 
specification  and  forecasts.  Note  that  GAIM-GM  can  also  be  run  with  no  data  ingested 
and  in  this  case  the  IFM  background  and  error  covariance  matrix  derived  from  a  set  of 
1107  IFM  runs  creates  an  ionospheric  climatology  output.  Thompson  (2006)  investigated 
such  impacts  due  to  data  variations  by  running  GAIM-GM  for  five  scenarios  with 
varying  combinations  of  both  GPS  TEC  data  and  ionosonde  electron  density  profiles. 
These  data  combinations  were  run  over  a  30  day  period  and  compared  to  TOPEX  TEC 
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measurements  and  also  compared  to  ionosonde  electron  density  profiles  from  Bear  Lake 
Observatory  in  Northern  Utah.  It  was  found  that  combining  GPS  and  ionosonde  data 
created  more  accurate  results  than  simply  using  one  type  of  data  and  not  another. 
Furthennore,  increasing  GPS  data  without  increasing  the  output  grid  spacing  may  have  a 
negative  impact  on  model  accuracy  ( Thompson  et  al.,  2006).  Thompson’s  work  did  not 
include  the  impact  of  DMSP  data  on  model  specification  accuracy.  Investigations  into 
forecast  accuracy  from  various  combinations  of  data  ingests  are  included  in  the 
recommendations  section.  Nevertheless,  in  an  effort  to  use  the  most  representative 
specification  for  model  accuracy  analysis,  GPS,  ionosonde  and  DMSP  are  used  in  the 
specification. 
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III.  Methodology 


This  chapter  discusses  the  methods  used  in  carrying  out  the  research.  Sections 
within  the  methodology  include:  (1)  specifying  the  geophysical  conditions  of  the  model 
cases,  (2)  running  the  IFM,  (3)  obtaining  input  data  and  reducing  it  into  useful  forms,  (4) 
choosing  the  data  locations  to  be  ingested,  (5)  running  GAIM-GM,  (6)  changing  the 
output  grid  to  focus  on  CONUS,  (7)  how  the  comparative  analysis  are  carried  out. 

3.1  Geophysical  conditions 

GAIM-GM  was  run  for  four  different  cases  with  varying  geophysical  conditions. 
The  dates,  geomagnetic  storming  conditions  (Kp  values),  and  solar  activity  (F10.7)  values 
for  each  case  are  set  forth  in  Table  1.  These  four  cases  produce  GAIM-GM  output  from 
geomagnetic  quiet  conditions  (low  Kp)  and  stonning  conditions  (high  Kp)  as  well  as 
solar  min  (low  F10.7)  and  solar  max  (high  F10.7).  Table  2  shows  the  relationship  between 
Kp  values  and  geomagnetic  stonning  conditions.  Geomagnetic  storms  alter  the 
ionosphere  by  enhancing  the  ionospheric  electrical  currents  via  incoming  energy  and 
plasma  ejected  from  the  sun.  An  enhancement  in  ion  densities  and  fluctuations  or  the 
disappearance  of  the  F2  peak  may  result  from  these  storms  and  thereby  impact  space 
based  capabilities. 


Table  1.  Geophysical  conditions  and  dates  for  the  four  cases  investigated. 


Case 

Days 

Year 

Geomagnetic  Activity  (Kp) 

Solar  Activity  (F10.7) 

1 

211-213 

2010 

Quiet  (2) 

Minimum  (84) 

2 

215-217 

2010 

Moderate  Storm  (6) 

Minimum  (83) 

3 

95-97 

2010 

Strong  Storm  (7) 

Minimum  (78) 

4 

170-172 

2001 

Quiet  (3) 

Maximum  (205) 
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Each  of  the  four  scenarios  was  run  to  produce  three  days  of  reliable  output.  The 
quiet  geomagnetic  conditions  during  solar  minimum  are  taken  from  days  211-213  (case 
1)  in  2010.  The  geomagnetic  storming  cases  are  for  days  215-217  (case  2)  and  days  95-97 
(case  3)  in  2010  and  correspond  to  moderate  and  strong  geomagnetic  storms  respectively. 
Quiet  geomagnetic  activity  during  solar  maximum  conditions  is  taken  from  days  170-172 
(case  4)  in  2001.  Case  1  and  case  2  run  nearly  consecutively.  This  is  by  design  as  both 
cases  are  therefore  taken  from  the  summer  time.  This  helps  to  eliminate  the  seasonal 
variation  associated  with  the  ionosphere.  Since  the  onset  time  of  geomagnetic  storms 
varies  from  storm  to  stonn,  two  geomagnetic  storms  were  investigated.  Case  2  and  case 
3  were  chosen  as  their  storm  onset  times  are  offset  by  ~9  hours.  This  approach  enhanced 
the  investigation  on  model  forecast  accuracy  during  geomagnetic  storms. 

Table  2.  Geomagnetic  storming  conditions  related  to  Kp  values,  adapted  from  SWPC  website 
(http  ://www.swpc.noaa.gov/NOAAscales/index.html#GeomagneticStorms) _ 


Kp  Index 

Storm  Strength 

5 

Minor 

6 

Moderate 

7 

Strong 

8 

Severe 

9 

Extreme 

3.2  Running  IFM 

Because  the  IFM  provides  the  background  upon  which  the  ionospheric 
perturbations  are  imposed  in  GAIM-GM,  the  IFM  must  be  run  and  this  background 
ionosphere  must  be  in  place  before  running  GAIM-GM.  The  input  parameters  needed  to 
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run  IFM  include  year,  Julian  day,  Kp  value,  F10.7  and  F10.7A  (81  day  F10.7  running 
average). 

The  IFM  requires  a  24  hour  warm  up  period  before  reliable  data  is  obtained  as  the 
IFM  uses  output  from  the  previous  day  to  generate  current  output.  Furthermore,  the  IFM 
must  be  run  for  a  sufficient  duration  as  the  forecast  output  for  GAIM-GM  likewise  need 
background  IFM  data  (i.e.  specification  and  forecasts  both  need  IFM  background). 

3.3  Data  Acquisition  and  Format 

With  the  background  IFM  in  place,  the  attention  turns  to  the  data  ingested  into 
GAIM-GM.  The  overall  process  can  be  summed  up  in  two  steps.  (1)  Obtaining  the  raw 
GPS,  ionosonde,  and  DMSP  data  and  (2)  fonnatting  the  data  to  a  user  friendly  version 
that  GAIM-GM  will  accept. 

3.3.1  GPS  Data 

Ingested  GPS  data  are  acquired  from  the  Continuously  Operating  Reference 
Station  (CORS)  database  maintained  by  NOAA.  This  data  is  packaged  as  Receiver 
Independent  Exchange  Format  (RINEX)  files.  Once  obtained,  the  files  are  stored  and 
formatted  such  that  GAIM-GM  can  ingest  the  time,  satellite  location,  ground  station 
location,  phase  TEC  and  phase  TEC  error. 

3.3.2  Ionosonde  Data 

Ionosonde  data  was  obtained  from  the  National  Geophysical  Data  Center  (NGDC) 
database  and  contains  raw  electron  density  profile  data.  This  raw  data  is  then  fonnatted 
into  a  data  text  file  containing  the  essential  components  for  the  electron  density  profile 
that  GAIM-GM  will  ingest.  The  formatted  data  includes  best  estimate  N111F2  and  hmF2, 
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uncertainties  in  those  parameters,  latitude  and  longitude  of  the  ionosonde  site  as  well  as 
electron  densities  for  select  altitudes  and  their  uncertainty. 

3.3.3  SSIES  Data 

The  SSIES  data  from  the  DMSP  satellites  was  obtained  from  the  NGDC  as  well 
as  provided  by  Dr.  Daniel  Ober  and  Dr.  Gordon  Wilson  from  the  Air  Force  Research 
Laboratory  (AFRL).  The  final  output  used  by  GAIM-GM  contains  the  time,  location, 
electron  density  and  uncertainty  of  the  measurements  taken  from  the  satellite. 

3.4  Data  Retention  Lists 

The  number  and  locations  of  ingested  data  can  be  specified  for  each  model  run. 
This  is  done  by  changing  the  station  retention  lists  located  within  the  directory  from 
which  GAIM-GM  is  run.  Both  specification  and  forecast  output  was  used  to  determine 
forecast  accuracy.  Therefore  it  was  necessary  to  run  GAIM-GM  several  times  for  each 
case,  each  with  different  station  retention  lists  in  order  to  obtain  the  most  representative 
forecasts  and  specifications  for  use  in  the  forecast  accuracy  analysis.  The  details  of  the 
GPS  and  ionosonde  lists  used  are  discussed  in  the  following  sections.  All  available 
SSIES  data  from  DMSP  satellites  FI 6,  F17  and  F18  were  used  in  cases  1-3  while  all 
SSIES  data  from  DMSP  satellites  FI 3,  F 14  and  F15  were  used  in  case  4.  The  model  was 
also  run  with  no  data  ingested,  and  results  from  these  runs  comprise  the  ionospheric 
reference  background  needed  for  the  skill  score  and  forecast  accuracy  analysis. 

3.4.1  AFWA  Data  List-Forecast 

To  closely  mimic  GAIM-GM  forecasts,  as  run  by  AFWA,  it  was  necessary  to 
obtain  the  lists  of  both  GPS  and  ionosonde  stations  that  AFWA  currently  uses  when 
running  the  model.  GAIM-GM  can  be  programmed  to  accept  data  from  many  locations 
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but  AFWA  only  uses  a  small  subset  of  the  total  available  data.  Lists  of  59  GPS  stations 
and  17  ionosonde  sites,  visualized  in  Figure  3,  ingested  by  AFWA  during  February  2011, 
were  used  in  the  GAIM-GM  model  runs  to  duplicate  AFWA  forecasts.  The  forecast 
output  generated  from  GAIM-GM  runs  from  these  lists  is  the  forecast  data  used  in  the 
computation  of  the  skill  score. 

3.4.2  All  Data  List-Global  Specification 

To  properly  evaluate  global  AFWA  forecast  grids  discussed  in  the  previous 
section,  a  globally  representative  specification  was  used.  This  specification  was  obtained 
from  a  list  of  351  GPS  sites  from  around  the  world,  generated  by  a  program  developed  at 
USU.  This  program  generates  a  global  distribution  of  ingested  GPS  sites.  The  ingested 
ionosonde  data  is  comprised  of  49  worldwide  ionosonde  sites  that  GAIM-GM  at  AFWA 
is  programmed  to  ingest.  Representations  of  these  data  sites  are  seen  in  Figure  4.  Recall 
that  only  17  of  these  sites  were  ingested  in  the  AFWA  forecast  run.  The  specifications 
from  these  runs  are  the  global  observations  used  in  detennining  global  skill  scores. 

3.4.3  AFWA  and  CONUS  Data  List  -  CONUS  Specification 

To  properly  analyze  GAIM-GM  forecast  output  over  CONUS,  a  representative 
CONUS  specification  was  used.  The  GPS  data  list  for  this  case,  visually  represented  in 
Figure  5,  includes  one  GPS  ground  station  for  each  grid  point  in  the  7  by  5  array  that 
encompasses  the  CONUS  area  with  30  of  the  35  grid  points  represented.  The  GPS 
ground  stations  chosen  over  CONUS  in  this  scheme  are  different  than  the  GPS  ground 
stations  used  in  the  AFWA  ingest  forecast  list  with  the  exception  of  Bermuda,  as  this 
location  was  used  in  both  cases  to  allow  greater  data  coverage  over  otherwise  data  sparse 
regions  in  the  Atlantic  Ocean.  GPS  ground  based  stations  outside  of  CONUS  are  identical 
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to  those  of  the  AFWA  data  ingest  list.  The  17  io nosonde  sites  from  the  AFWA  list  were 
used  in  these  runs. 

3.5  Running  GAIM-GM 

Once  the  IFM  runs  are  completed  and  the  data  is  retrieved  and  formatted,  GPS 
satellite  and  ground  based  station  biases  must  be  downloaded.  The  biases,  which  may 
stem  from  hardware  limitations  and  calibration  errors,  and  the  uncertainty  in  the  biases, 
are  given  in  nanoseconds.  Correcting  for  these  biases  increases  the  level  of  accuracy  of 
the  TEC  measurements  obtained  from  GPS. 

The  next  step  is  to  ensure  that  the  configuration  files  that  specify  the  start  time, 
duration,  forecast  increments  and  range  of  the  output  as  well  as  the  desired  data  to  be 
ingested  are  accurate.  GAIM-GM  can  generate  forecasts  valid  from  15  minutes  to  24 
hours  from  the  current  specification  in  15  minute  increments. 

It  is  important  to  understand  how  and  in  what  order  GAIM-GM  output  is 
generated.  Specifications  are  generated  from  the  IFM  background  and  the  data  ingested 
via  the  Kalman  filter.  A  forecast  suite  is  then  generated  from  each  specification.  For  the 
runs  completed  in  this  study,  3,  6,  12  and  24  hour  forecasts  are  obtained  from  each 
specification  and  saved  in  directories  that  contain  only  those  hourly  forecasts  (3-hour 
forecasts  are  saved  together  and  6-hour  forecasts  are  saved  together  and  so  forth). 
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(a)  GPS  Sites 


(b)  Ionosonde  Sites 

Figure  3.  AFWA  data  ingest  sites:  (a)  GPS  sites  (59),  (b)  ionosonde  sites  (17).  The  forecast  output 
used  in  the  accuracy  analysis  is  obtained  from  running  the  model  with  these  sites  ingested  into  the 
model. 
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(a)  GPS  Sites 


(b)  Ionosonde  Sites 

Figure  4.  All  data  ingest  sites:  (a)  GPS  sites  (351),  (b)  ionosonde  sites  (49).  The  global  specification 
output  used  in  the  accuracy  analysis  is  obtained  from  running  the  model  with  these  sites  ingested  into 
the  model. 
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(a)  GPS  Sites 


(b)  CONUS  GPS  Sites 


Figure  5.  The  AFWA  plus  CONUS  data  ingest  sites:  (a)  GPS  sites  (79),  (b)  CONUS  GPS  sites  (30). 
The  CONUS  specification  output  used  in  the  accuracy  analysis  is  obtained  from  running  the  model 
with  these  sites  ingested  into  the  model. 
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Once  all  the  forecasts  generated  from  a  specification  are  saved,  the  model  moves 
forward  15  minutes  to  the  next  specification  and  generates  the  next  suite  of  forecast 
outputs.  The  forecast  output  files  contained  within  their  respective  directories  are  the 
forecasts  valid  at  the  time  indicated  on  their  file  name.  The  file  naming  convention 
includes  the  letters  GMF,  followed  by  the  year,  Julian  day,  hour,  minute  and  netCDF  file 
extension,  for  example,  GMF201021 10000.NC.  This  file  would  be  valid  at  0000UT  on 
day  211,  year  2010.  A  24-hour  forecast  generated  from  a  specification  at  a  certain  time  is 
valid  24  hours  from  the  specification  time.  A  forecast  variable  is  included  in  the  netCDF 
file  that  indicates  the  number  of  seconds  out  from  the  specification  for  which  the  forecast 
is  valid.  Thus  a  forecast  variable  indicating  86400  seconds  is  a  24-hour  forecast. 

Recall  that  GAIM-GM  was  run  for  four  cases  of  various  geomagnetic  and  solar 
activity.  Each  case  was  run  four  times  giving  output  corresponding  to:  forecasts  obtained 
from  the  AFWA  retention  list,  global  and  CONUS  specifications  obtained  from  the  All 
and  AFWA  plus  CONUS  retention  lists  respectively,  and  reference  output  obtained  from 
running  the  model  with  no  data  ingested. 

3.6  Statistical  Analysis 

Several  different  types  of  statistics  and  plots  were  generated  to  carry  out  the 
analysis  of  GAIM-GM  forecast  accuracy.  These  include  plots  of  TEC  and  TEC  percent 
difference  as  well  as  forecast  mean  percent  differences,  RMSE  and  skill  score  values. 

3.6.1  TEC  and  TEC  Percent  Difference  Plots 

TEC  plots  and  forecast  TEC  percent  difference  plots  are  the  building  blocks  from 
which  all  other  plots  and  statistics  are  derived  in  this  study.  The  graphical 
representations  of  TEC  are  taken  directly  from  the  TEC  output  from  GAIM-GM.  The 
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forecast  TEC  percent  differences  are  computed  by  fonnula  5  where  the  differences  are 
taken  between  corresponding  grid  points  from  the  forecast  and  specification  TEC  outputs. 
Therefore,  forecast  TEC  percent  difference  plots  maintain  the  dimensions  of  the  arrays 
from  which  they  are  derived.  Furthermore,  only  one  forecast  TEC  percent  difference  can 
be  computed  for  each  time  step.  An  example  percent  difference  plot  is  shown  in  Figure 
6.  Positive  percent  errors  are  denoted  in  red  and  those  with  a  negative  error  are  indicated 
in  blue. 


/ 

Percent  Difference  =  (■ 


Forecast  —  Specif  ication^ 
Specification  ) 


*  100 


(5) 


Day  211  0000UT,  2010 


Percent 


Figure  6.  A  forecast  TEC  percent  difference  plot.  Each  grid  value  is  computed  via  equation  5, 
comparing  corresponding  forecast  and  specification  grid  points.  Positive  and  negative  percent 
differences  are  shown  in  red  and  blue  respectively. 
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3.6.2  Mean  Percent  Difference 


Mean  percent  difference  (MPD)  and  mean  absolute  percent  difference  (MAPD) 
values  were  obtained  by  averaging  the  grid  values  from  the  forecast  TEC  percent 
difference  arrays,  such  as  those  in  Figure  6.  The  MPD  simply  averages  the  values  of  all 
grids  within  a  forecast  TEC  percent  difference  plot.  The  MAPD  takes  the  absolute  value 
of  each  grid  point  in  the  forecast  TEC  percent  different  plot  and  computes  the  average  of 
those  values.  The  MPD  and  MAPD  can  be  computed  only  once  for  each  forecast  TEC 
output.  The  MPD  and  MAPD  were  plotted  together  as  line  charts  to  depict  the  total 
percent  difference  as  well  as  the  positive  and  negative  variations  in  the  differences. 

3.6.3  RMSE  and  Skill  Score  Value 

Since  the  global  output  for  the  forecasts,  specifications  and  reference  are  the  same 
size  (44  by  24  arrays),  the  global  skill  scores  were  straightforward  to  compute.  The  same 
can  be  said  of  the  CONUS  skill  scores  as  the  grid  sizes  for  the  components  of  the  skill 
score  were  all  7  by  5  arrays. 

Skill  scores  were  generated  for  each  forecast  output  (3-hour,  6-hour,  12-hour  and 
24-hour)  for  each  of  the  four  cases.  The  skill  scores  were  computed  using  MATLAB  and 
in  accordance  with  the  formulas  specified  in  the  background  section. 

GAIM-GM  generates  output  every  15  minutes  resulting  in  96  outputs  per  day. 
Each  case  encompasses  three  days,  providing  288  time  increments  from  which  skill 
scores  were  obtained  and  plotted.  The  forecast  skill  score  shows  how  well  the  forecast 
compares  to  a  reference.  Recall  that  positive  skill  scores  indicate  an  improvement  over 
the  reference  whereas  negative  skill  scores  indicate  that  the  reference  was  more 
representative  of  the  ionosphere.  It  is  important  to  understand  that  a  negative  skill  score 
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does  not  necessarily  indicate  that  the  forecast  was  poor  or  inaccurate,  only  that  the 
reference  did  a  better  job  at  specifying  the  ionosphere  than  did  the  forecast.  It  is  possible 
to  obtain  excellent  forecasts  and  negative  skill  score.  It  is  also  possible  to  obtain  great 
errors  in  forecast  output  but  maintain  high  skill  scores.  It  is  the  ratio  of  the  two  RMSEs 
that  indicate  the  skill  score  and  not  the  absolute  accuracy  of  the  forecast. 

The  RMSE  and  RMSE  reference,  the  building  blocks  of  the  skill  score,  were  also 
plotted  to  show  the  overall  error,  trends  in  the  error  as  well  as  the  relationship  between 
RMSE  and  skill  score. 
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IV.  Results  and  Analysis 


The  objective  of  this  research  was  to  determine  GAIM-GM  model  forecast 
accuracy.  This  chapter  presents  the  results  and  analysis  of  this  investigation  through  the 
use  of  the  skill  scores  and  other  statistics  obtained  from  running  GAIM-GM  over  various 
geophysical  conditions.  Four  cases  were  investigated,  first  was  a  geomagnetic  quiet, 
solar  minimum  case.  The  next  two  contained  geomagnetic  storming  conditions  during 
solar  minimum.  The  fourth  was  during  geomagnetic  quiet,  solar  maximum  conditions. 
This  chapter  analyzes  forecast  accuracy  over  both  the  entire  globe  and  CONUS. 

4.1  Global  Forecast  Trends  and  CONUS  Focus 

The  global  TEC  forecasts  were  analyzed  via  the  skill  score  and  the  forecast 
RMSE.  A  sample  of  4608  values  was  obtained  when  combining  the  four  cases,  three 
days  of  output,  the  four  forecast  periods  (3,  6,  12,  24)  and  the  output  cadence  of  15 
minutes.  Of  these  4608  values,  only  twice  were  negative  skill  scores  obtained.  These 
occurred  for  the  3-hour  forecast  output  at  21 15UT  and  2130UT  on  day  95,  2010,  case  3, 
with  values  of  -0.0453  and  -0.0048,  respectively.  The  abundant  positive  skill  scores 
show  that  globally,  the  forecast  is  virtually  always  more  accurate  than  the  reference. 

The  3-hour  global  forecast  skill  scores  and  RMSEs  for  case  1  are  plotted  and 
shown  in  Figure  7.  Although  there  are  minor  fluctuations  in  the  skill  score,  the  forecast 
RMSE  and  the  reference  RMSE  shown  in  green,  red  and  blue  respectively,  the  overall 
variability  is  low.  For  example,  the  forecast  RMSE  range  over  the  three  days  is  1.36 
TECU  with  minimum  and  maximum  forecast  RMSE  values  of  1.23  TECU  and  2.59 
TECU  respectively.  A  typical  TECU  forecast  range  for  this  case  is  from  0  to  40  TECU. 
The  range,  average,  and  standard  deviation  of  the  3-hour  forecast  skill  scores  and  forecast 
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RMSE  for  case  1  are  included  in  Table  3.  Similar  results  denoting  low  variability  were 
obtained  for  cases  2-4. 


Table  3.  Range,  average,  and  standard  deviation  for  the  3-hour  global  forecast  skill  score  and 
RMSE  taken  from  case  1. 


Global  3-hour  Forecast  Case  1 

Skill  Score 

Forecast  RMSE  (TECU) 

Range 

0.33 

1.36 

Average 

0.33 

1.86 

Standard  Deviation 

0.06 

0.28 

Global  TEC  forecasts  show  little  variability  in  skill  score  and  RMSE.  This  is 
likely  due  to  the  inclusion  of  data  sparse  areas  such  as  the  oceans  and  Africa  that  result  in 
the  forecast  output  reverting  back  to  the  specification  background.  It  should  be  noted  that 
high  latitudes,  those  greater  than  60  degrees,  were  omitted  when  determining  global 
forecast  accuracy  as  GAIM-GM  does  not  assimilate  data  from  these  regions.  Global 
output  also  masks  the  full  extent  of  diurnal  impacts  and  the  impact  of  the  Kalman  filter  in 
enhancing  forecast  accuracy.  Therefore  in  order  to  determine  the  efficiency  of  the 
Kalman  filter  as  well  as  the  extent  to  which  GAIM-GM  can  account  for  the  diurnal 
pattern  of  the  ionosphere,  the  data-rich  CONUS  area  was  selected  for  the  remainder  of 
the  analysis. 

4.2  Hourly  Forecast  Trends  and  3-Hour  Forecast  Focus 

GAIM-GM  can  forecast  up  to  24  hours  from  the  current  specification.  The  3- 
hour,  6-hour,  12-hour  and  24-hour  forecasts  were  generated  as  part  of  this  analysis.  In 
reviewing  the  skill  scores  from  these  forecasts  for  each  of  the  four  cases,  some  overall 
trends  were  evident.  Foremost  among  these  was  the  consistent  trend  that  the  3 -hour  skill 
scores  outperformed  all  others.  Figure  8  shows  the  CONUS  skill  scores  for  the  four 
forecast  periods  obtained  from  case  1 . 
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Days  211-213,2010 


- Forecast  RMSE  (TECU) - Reference  RMSE  (TECU) - Ski  II  Score - Kp 


Figure  7.  Global  forecast  skill  score  (green),  forecast  RMSE  (red)  and  reference  RMSE  (blue)  taken 
from  the  3-hour  forecasts  for  case  1,  days  211-213, 2010. 


The  3 -hour  forecast  skill  score,  denoted  in  green  is  consistently  greater  than  that 
of  the  other  three  forecasts.  This  is  further  illustrated  in  Figure  9,  a  bar  chart  of  the 
average  skill  scores  and  average  RMSE  broken  up  by  forecast  period  for  all  cases.  Figure 
9  indicates  that  as  forecast  range  increases  from  3  hours  to  24  hours,  the  skill  scores 
decrease  and  the  RMSE  increases.  The  average  3-hour  forecast  skill  scores  from  case  4, 
for  example,  decreases  from  0.51  to  0.28.  The  greatest  RMSE  difference  is  also  found  in 
case  4  when  the  RMSE  increases  from  a  3-hour  forecast  output  minimum  of  3.86  TECU 
to  5.85  TECU  at  24  hours.  The  high  skill  scores  and  high  RMSE  seen  in  case  4  are  the 
result  of  the  greater  differences  between  the  forecast  and  reference  RMSE  when 
compared  to  the  other  three  cases.  What  is  consistent  however  is  that  in  the  cases 
investigated,  the  average  3-hour  forecast  skill  scores  were  greatest  when  compared  to  the 
other  forecast  periods.  Furthermore,  the  3-hour  forecast  RMSE  were  the  least  of  the  four 
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For  this  reason  the  3 -hour  forecast  output  was  chosen  to  be  analyzed  for  the  remainder  of 
the  study. 

4.3  Case  1  Results 

The  baseline  run  for  this  study  was  taken  from  days  211-213  of  year  2010.  This 
mid-summer  time  period  was  characterized  by  quiet  geomagnetic  conditions  during  solar 
minimum.  Also,  a  C3  solar  flare  was  observed  at  0826UT  on  day  213,  but  it  did  not 
impact  CONUS  data. 

4.3.1  CONUS  Skill  Score 

The  3-hour  forecast  CONUS  skill  scores  for  case  1  are  plotted  in  green  on  Figure 
10  along  with  the  Kp  values.  The  three  day  plot  shows  a  decrease  in  skill  score  on  each 
of  the  three  days,  centered  near  06UT.  The  lowest  skill  score  occurs  on  day  213  near 
6UT  with  values  as  low  as  -0.42. 


Days  211-213,  2010 


—  3  hr  skill  score - 6  hr  skill  score - 12  hr  skill  score  24  hr  skill  score - Kp/10 


Figure  8.  Forecast  skill  scores  for  all  forecast  periods  obtained  for  case  1,  days  211-213,  2010. 
Trending  patterns  are  consistent  among  the  four  forecast  ranges.  Overall,  the  3-hour  forecast  skill 
scores  (green)  are  greater  than  the  forecast  skill  scores  obtained  from  the  others  three  forecast 
periods. 
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(b)  RMSE 

Figure  9.  Bar  charts  showing  (a)  average  skill  scores  for  the  four  forecast  periods  for  all  cases  and 
(b)  average  RMSE  for  all  four  forecast  periods  for  each  case. 
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Average  skill  scores  were  computed  for  both  daytime  (1300UT-2215UT)  and  nighttime 
(0330UT-0745UT)  and  are  0.49  and  0.06  respectively.  The  overall  average  skill  score 
for  the  3-hour  forecast  TEC  output  is  0.39. 

The  above  average  daytime  skill  score  and  the  below  average  skill  score  at  night 
indicates  a  diurnal  trend  in  forecast  accuracy.  The  RMSE  was  plotted  to  determine 
overall  forecast  error,  to  discern  the  trending  in  errors,  and  to  further  investigate  the 
diurnal  impact  on  forecast  accuracy. 


Days  211-213,  2010 


—  3  hr  skill  score  - Kp/10 


Figure  10.  3-hour  forecast  skill  scores  and  Kp  values  for  case  1,  days  211-213,  2010.  Kp  values  are 
reduced  by  a  factor  of  ten  to  fit  within  the  range  of  the  plot.  Note  the  decrease  in  skill  score  at  night 
at  06UT  for  each  of  the  three  days. 


4.3.2  CONUS  RMSE 

Forecast  and  reference  TEC  RMSE  for  the  3-hour  forecast  from  case  1  were 
plotted  in  conjunction  with  the  skill  score  and  Kp  values  and  are  included  in  Figure  11. 
The  forecast  and  reference  RMSE,  denoted  by  red  and  blue  lines  respectively,  reach 
minima  at  -06UT  each  day,  consistent  with  the  decrease  in  skill  score  seen  in  the  green 
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lines  in  10.  Forecast  and  reference  RMSE  therefore  appear  to  have  a  diurnal  pattern.  The 
forecast  and  reference  RMSE  not  only  increase  during  the  daytime,  but  the  difference 
between  the  forecast  and  reference  RMSE  also  increases.  Additionally,  the  range  of  the 
reference  RMSE  is  greater  than  the  range  of  the  forecast  RMSE.  Both  RMSEs  remain 
near  1  TECU  at  night  but  the  reference  RMSE  peaks  at  ~6  TECU  on  day  213  while  the 
forecast  RMSE  peaks  at  ~3  TECU  during  the  daytime  on  day  213.  This  is  the  reason  for 
the  rise  in  skill  scores  during  the  day,  as  the  skill  score  is  a  function  of  the  ratio  of  the 
forecast  RMSE  to  the  reference  RMSE  and  not  an  indicator  of  the  overall  error  associated 
with  the  forecast. 

Days  211-213,  2010 


UT  Hour 


- Forecast  RMSE  (TECU) - Reference  RMSE  (TECU) - Skill  Score  - Kp  | 

Figure  11.  Forecast  and  reference  TEC  RMSE  as  well  as  skill  score  and  Kp  values  for  3-hour 
CONUS  forecasts  from  case  1.  Where  the  skill  score  approaches  zero  is  where  the  RMSEs  are 
closest.  The  highest  skill  scores  are  found  where  the  difference  between  the  forecast  and  reference 
RMSE  is  the  greatest. 


4.3.3  CONUS  MPD 

The  Mean  Percent  Difference  and  Mean  Absolute  Percent  Difference  (MAPD) 
between  the  forecasts  and  the  specifications  were  computed.  Whereas  the  RMSE  is  a 
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measure  of  the  error,  the  MPD  plots  indicate  the  percent  difference  between  the  forecast 
and  the  specification.  The  MPD  and  MAPD  were  plotted  together  in  blue  and  red 
respectively.  Figure  12  shows  the  MPD  which,  like  skill  score  and  RMSE,  also  shows  a 
diurnal  trend  with  negative  values  at  night  and  positive  values  during  the  day.  The 
average  daytime  MPD,  taken  from  the  same  time  frame  as  the  RMSE  computation 
presented  earlier  (1300UT-2215UT),  is  14.9%. 

Days  21 1-213,  2010 


- MAPD  (%)  - MPD  (%)  - Kp’10 


Figure  12.  Mean  Absolute  Percent  Difference  (MAPD  in  red)  and  Mean  Percent  Differences  (MPD 
in  blue)  for  case  1,  days  211-213,  2010.  Note  the  distinct  diurnal  trend,  with  MPD  values  negative  at 
night  (0330UT-0745UT)  and  positive  during  the  day  (1300UT-2215UT). 

The  nighttime  average  MPD  (0330UT-0745UT)  is  -12.3%.  This  indicates  a  negative 
forecast  bias  at  night  and  a  positive  forecast  bias  during  the  day.  A  positive  bias  indicates 
that  the  forecast  output  was  greater  than  the  specification  output  whereas  a  negative  bias 
indicates  that  the  specification  TEC  values  were  greater  than  the  forecast  TEC  values. 

The  daytime  peak  MAPD  for  days  211  and  212  are  25%,  but  the  peak  on  day  213 

is  noticeably  higher  at  above  40%.  To  investigate  this,  the  F0F2  critical  frequency  for 
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select  CONUS  locations  for  the  same  time  frame  is  included  in  Figure  13.  The  F0F2 
parameter  indicates  the  maximum  frequency  reflected  by  the  ionosphere  and  is  therefore 
a  good  indicator  of  the  strength  of  the  ionosphere  and  consequently  of  TEC.  Higher 
reflected  frequencies  indicate  increased  plasma  densities  and  therefore  a  stronger 
ionosphere.  The  slight  decrease  in  F0F2  on  day  213  indicates  that  the  ionosphere  was 
likely  weaker  than  the  previous  day.  A  greater  forecast  TEC  value  than  observation  TEC 
value  would  result  in  an  enhancement  in  the  positive  bias  and  this  shows  up  on  the  MPD 
plots  for  day  213. 

4.4  Case  2  Results 

Case  2  includes  the  three  day  period  for  days  215-217,  2010.  Beginning  two  days 
after  the  end  of  case  1,  this  case  is  also  during  mid-summer.  A  geomagnetic  storm  with 
Kp  values  above  6  peaked  at  22UT  on  day  215.  The  diurnal  bias  calculations  for  case  2 
are  from  the  same  time  range  as  those  in  case  1  as  there  is  not  a  significant  change  in  day 
and  night  durations  between  these  two  cases. 

4.4.1  CONUS  Skill  Score 

Skill  scores  for  this  case  decrease  during  night  as  in  case  1 .  Negative  skill  scores 
are  observed  at  night  for  each  of  the  three  days  in  case  2.  A  negative  spike  in  skill  scores 
is  also  observed  near  22UT  on  day  215,  coincident  with  the  peak  in  the  storm  and  during 
daytime.  This  is  shown  on  Figure  14,  the  plot  of  skill  scores  for  case  2.  The  decrease  at 
22UT  on  day  215  is  the  result  of  the  forecast  RMSE  being  greater  than  the  reference 
RMSE. 
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Figure  13.  FoF2  critical  frequency  plots  for  select  CONUS  locations  during  case  1,  days  211-213, 
2010. 


4.4.2  CONUS  RMSE 

The  RMSE  plot  for  case  2,  Figure  15,  illustrates  the  diurnal  pattern  of  RMSE 
increasing  during  the  day  and  decreasing  at  night.  It  also  shows  the  tendency  for  skill 
scores  to  decrease  at  night  as  the  forecast  and  reference  RMSE  not  only  decrease,  but 
become  close  to  one  another.  In  this  particular  case  however,  there  are  several  deviations 
from  what  was  observed  in  case  1,  namely  the  overall  increase  in  the  magnitude  of  the 
RMSEs  and  the  extreme  peak  in  forecast  RMSE  in  line  with  the  peak  of  the  geomagnetic 
storm  at  22UT  on  day  215. 

When  compared  to  case  1,  where  the  average  forecast  RMSE  was  1.42  TECU, 
case  2  average  forecast  RMSE  is  2.27  TECU,  or  a  59%  increase.  The  forecast  RMSE 
range  more  than  doubles  in  case  2  from  case  1,  going  from  0.34  TECU  -  3.87  TECU  for 
case  1  to  0.26  TECU  -  8.36  TECU  in  case  2. 
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Days  215-217,  2010 


UT  Hour 


- 3  hr  skill  score -  Kp/10 


Figure  14.  Skill  scores  and  Kp  values  for  case  2,  days  215-217,  2010.  Nightly  decreases  in  skill  score 
are  present,  as  they  were  in  case  1. 


Days  215-217,  2010 


UT  Hour 


- Forecast  RMSE(TECU)  - Reference  RMSE  (TECU) - Ski  II  Score  - Kp 


Figure  15.  Forecast  and  reference  RMSE  as  well  as  skill  scores  and  Kp  values  for  case  2,  days  215- 
217,  2010.  Diurnal  peak  in  RMSE  is  in  line  with  the  peak  of  the  geomagnetic  storm. 


4.4.3  CONUS  MPD 

The  MPD  plots  for  case  2  also  indicate  a  diurnal  bias  to  under  forecast  at  night 
and  over  forecast  during  the  day.  The  negative  nighttime  MPD  values  are  seen  in  case  2 
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in  Figure  16  as  they  were  in  Figure  12  for  case  1.  The  average  MPD  at  night  is  -7.6% 
compared  to  25.0%  during  the  day. 


Days  215-217,  2010 


- MAPD  (%)  - MPD  (%) - Kp'10 


Figure  16.  Forecast  TEC  MPD  and  MAPD  values  for  case  2,  days  215-217,  2010. 

The  geomagnetic  storm  impacted  the  average  and  ranges  of  the  MAPD.  The 
average  MAPD  increased  from  17.2%  for  case  1  to  22.6%  for  case  2,  a  5.4%  increase. 
The  max  MAPD  also  increased  from  42.7%  to  56.1%. 

The  peak  of  the  geomagnetic  storm  coincided  with  peak  daytime  ionization  over 
CONUS  during  case  2.  Case  3  looks  at  a  geomagnetic  stonn  that  peaks  in  the  morning 
over  CONUS. 

4.5  Case  3  Results 

Case  3  is  taken  from  days  95-97  of  year  2010.  This  time  frame  was  chosen  as  it 
too  occurs  during  a  solar  minimum  with  a  geomagnetic  storm.  The  main  differences 
from  case  2  are  that  the  timing  of  the  storm  is  offset  from  daytime  peak  ionization,  and 
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the  storm  is  stronger  than  case  2.  This  second  case  was  selected  to  more  fully  investigate 
the  influence  of  the  timing  of  a  geomagnetic  storm  on  forecast  accuracy  as  well  as  the 
extent  of  the  diurnal  influence  and  bias  in  the  model. 

4.5.1  CONUS  Skill  Score 

The  skill  score  plot  for  case  3,  Figure  17,  shows  negative  skill  scores  at  night  on 
the  first  day  and  during  peak  daytime  ionization  on  the  first  day,  22UT  day  95,  while  the 
Kp  values  remained  elevated  (Kp  of  4).  This  is  similar  to  what  was  observed  in  both 
cases  1  and  2  as  skill  scores  decrease  and  at  times  become  negative  either  at  night  when 
the  forecast  and  reference  RMSE  were  relatively  low  or  during  peak  daytime  ionization 
during  geomagnetic  storming. 

4.5.2  CONUS  RMSE 

The  RMSE  plot  for  case  3,  Figure  18,  shows  that  the  errors  increase  and  reach  a 
peak  during  peak  daytime  ionization.  Peak  ionization  in  case  3  is  offset  from  the  peak  in 
the  geomagnetic  stonn,  indicated  by  the  max  Kp  value  at  ~l  OUT  on  day  95.  When 
compared  to  case  2,  the  average  forecast  RMSE  for  case  3  increases  from  2.27  TECU  to 
2.62  TECU,  a  15%  increase. 

4.5.3  CONUS  MPD 

The  MPD  and  MAPD  plot  for  case  3,  Figure  19,  reveals  that  the  greatest  MAPD 
and  MPD  occur  during  daytime  as  it  was  for  case  2.  The  increased  strength  in  the  storm 
may  have  influenced  the  maximum  MAPD  values.  The  max  MAPD  value  for  case  3 
was  88%  at  22UT  on  day  95.  The  maximum  MAPD  for  case  2  was  56%  at  1830UT  on 
day  216,  over  30%  lower  than  that  for  case  3. 
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Days  95-97,  2010 


UT  Hour 


—  3  hr  skill  score -  Kp/10 


Figure  17.  Case  3  skill  scores,  from  days  95-97,  2010. 


Days  95-97,  2010 


UT  Hour 


- Forecast  RMSE(TECU)  - Reference  RMSE  (TECU) - Skill  Score  - Kp 


Figure  18.  Forecast  and  reference  RMSE  and  skill  scores  for  case  3,  days  95-97,  2010.  Note  the 
offset  in  max  storm  time  (10UT  day  95)  and  peak  ionization  (22UT  day  95).  The  forecast  RMSE 
(red)  is  greater  than  the  reference  RMSE  at  22UT  day  95  during  storming  conditions,  similar  to  case 
2. 


As  with  the  previous  two  cases,  the  MPD  also  shows  a  diurnal  bias.  Whereas  the 
first  two  cases  exhibit  the  tendency  to  have  negative  errors  at  night,  case  3  MPD  decrease 
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at  night,  but  may  not  become  negative,  as  is  seen  on  day  96  where  all  MPD  values  are 
positive.  Further  evidence  of  this  is  found  when  taking  the  average  nighttime  MPD  for 
case  3  (14.3%)  and  comparing  is  to  the  equivalent  results  for  case  1  (-12.3%)  and  case  2 
(-7.6%)  where  the  latter  two  are  negative.  The  night  timeframe  was  taken  between 
0215UT  and  0900UT  as  case  3  is  in  springtime. 

4.6  Case  4  Results 

Case  4  encompasses  a  three  day  period  from  day  170-172  in  2001.  This  time 
frame  was  chosen  as  it  corresponds  to  low  geomagnetic  activity,  similar  to  case  1,  but 
occurs  during  a  solar  maximum  period  when  TEC  values  are  greater  when  compared  to 
solar  minimum.  It  is  also  during  the  summer  solstice  when  the  Northern  Hemisphere 
attains  maximum  solar  radiation. 


Days  95-97,  2010 


- MAPD  (%)  - MPD  (%)  - Kp*10 


Figure  19.  MPD  and  MAPD  for  case  3,  days  95-97,  2010.  Note  the  peak  MAPD  of  88%  on  day  95. 
Nighttime  MDP  (0215UT-0900UT)  values  decrease,  but  are  not  necessarily  negative  as  they  were  fore 
cases  1  and  2. 
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4.6.1  CONUS  Skill  Score 


Case  4  skill  scores  are  included  in  Figure  20.  They  too  show  a  tendency  to 
increase  and  decrease  throughout  the  day.  The  minimum  skill  score  values  are  offset  in 
case  4  from  those  in  the  previous  three  cases.  The  decreases  are  typically  seen  near 
10UT.  This  time,  during  the  summer  solstice,  corresponds  to  sunrise  over  CONUS  which 
is  from  0700UT  to  1245UT.  Recall  that  the  negative  skill  scores  for  the  previous  three 
cases  occurred  near  06UT. 

4.6.2  CONUS  RMSE 

A  diurnal  pattern  is  present  during  solar  max,  as  it  was  during  solar  min,  with  the 
minimum  RMSE  attained  at  sunrise.  Because  the  ratio  of  the  reference  RMSE  to  the 
forecast  RMSE  is  greater  during  the  daytime  than  was  seen  in  the  previous  three  cases, 
the  overall  skill  score  for  case  4  is  the  highest  of  the  four  cases  at  0.51.  The  average 
forecast  RMSE  is  greatest  in  this  case  at  3.85  TECU  compared  to  1.43  TECU,  2.26 
TECU,  2.61  TECU  for  cases  1  through  3  respectively. 


Days  170-172,  2001 


UT  Hour 

—  3  hr  skill  score  ■  1  Kp/1 0 1 


Figure  20.  Case  4  skill  scores  and  Kp  values  for  days  170-172,  2001. 
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Days  170-172,  2001 


UT  Hour 


- Forecast  RMSE  (TECU)  - Reference  RMSE  (TECU) - Skill  Score  - Kp 


Figure  21.  Case  4  forecast  and  reference  RMSE  values  as  well  as  Kp  and  skill  scores  for  days  170- 
172,2001. 


4.6.3  CONUS  MPD 

The  MPD  and  MAPD  fore  case  4,  Figure  22,  show  daily  peaks  for  both  MPD  and 
MAPD  at  ~1 3UT  on  each  of  the  three  days.  Daytime  and  nighttime  for  case  4  occur 
between  1245UT  and  2230UT  and  0400UT  and  0700UT  respectively.  The  average 
nighttime  MPD  is  3.9%  and  the  daytime  MPD  is  13.2%.  Once  again,  the  MPD  is  greater 
during  the  day  than  at  night. 

4.7  Diurnal  Comparisons  for  All  Cases 

Figure  23,  bar  charts  that  show  the  average  daytime  and  nighttime  MPD  and 
RMSE  for  each  case,  illustrates  the  diurnal  trend  in  each  of  the  four  cases.  The  nighttime 
(blue)  bars  are  less  than  the  daytime  (red)  bars  in  every  case.  Furthermore,  in  every  case, 
the  nighttime  values  are  less  than  the  overall  average  while  the  daytime  values  are  greater 
than  the  overall  values. 
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Days  170-172,  2001 


- MAPD  (%)  - MPD  (%)  - Kp*10 


Figure  22.  MPD  and  MAPD  for  case  4,  days  170-172,  2001. 

4.8  Forecast  Accuracy  for  Kp  Values 

There  were  1152  forecasts  taken  from  the  four  cases  of  the  3-hour  forecasts. 
Average  RMSE  were  plotted  by  Kp  value  for  this  data  set  and  the  results  are  included  in 
Figure  24a.  With  the  exception  of  Kp  values  of  5,  the  average  RMSE  increases  as  Kp 
increases.  Results  for  Kp  values  of  8  and  9  are  absent  as  the  greatest  Kp  value  reached 
among  the  four  cases  was  7.  Figure  24b  indicates  the  number  of  occurrences  for  each 
Kp.  Numbers  of  occurrences  decreased  as  Kp  increased. 

Finally,  the  average  RMSE  and  Kp  correlation  was  broken  up  into  two  categories, 
quiet  and  storming,  given  by  Kp  values  of  0-3  and  4-9  respectively.  These  are  plotted  as 
scatter  plots  in  Figure  25.  For  such  plots,  correlation  increases  as  the  distribution  of  the 
points  on  the  scatter  plot  becomes  less  diffuse  (more  tightly  packed  around  a  common 
trend  line).  Figure  25a,  geomagnetic  quiet  conditions,  is  more  diffuse  and  therefore  has 
less  correlation  than  geomagnetic  storming  conditions,  Figure  25b. 
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Figure  23.  Diurnal  comparison  of  (a)  MPD  and  (b)  RMSE  for  all  cases.  In  each  case  the  nighttime 
values  are  less  than  the  overall  average  values  and  the  daytime  values  are  greater  than  the  overall 
average. 
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Figure  24.  (a)  Average  RMSE  values  broken  up  by  Kp  value  and  (b)  number  of  occurrences  of  each 
Kp  value  for  the  1152  forecasts  over  the  four  cases.  Kp  values  of  7  were  the  highest  observed  during 
the  four  cases  and  therefore  Kp  values  of  8  and  9  are  not  represented. 
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Figure  25.  Scatter  plot  of  the  average  TEC  values  and  average  RMSE  for  geomagnetic  quiet 
conditions  (a)  and  storming  conditions  (b).  Geomagnetic  storming  shows  greater  correlation  than 
geomagnetic  quiet  conditions,  illustrated  by  a  less  diffuse  distribution  of  data  points. 
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V.  Conclusions  and  Recommendations 


The  two  parts  of  this  final  chapter  include  a  summary  with  final  conclusions  of 
GAIM-GM  accuracy  as  well  as  recommendations  for  future  work  investigating  GAIM- 
GM  model  accuracy. 

5.1  Conclusions 

Knowing  the  limitations  and  accuracy  of  any  forecast  model  is  necessary  in 
putting  the  proper  level  of  confidence  into  the  forecast.  GAIM-GM  model  forecast 
accuracy  was  analyzed  by  comparing  forecasts  and  specifications  to  a  reference 
background  ionosphere  derived  from  IFM  output  and  error  covariance  matrix  of  the 
GAIM-GM  Kalman  filter.  This  was  done  by  looking  at  model  accuracy  for  four  cases, 
corresponding  to  changes  in  geomagnetic  storming  conditions  and  solar  activity 
variations. 

When  looking  at  the  percent  of  positive  skill  scores,  the  skill  scores,  RMSE  and 
MAPD  for  the  forecasts  periods  averaged  over  the  four  cases,  the  3 -hour  forecast 
accuracy  is  greatest  for  all  statistical  parameters.  Moreover,  the  forecast  accuracy 
degrades  as  the  forecast  range  increases.  The  forecast  accuracy  typically  decreases  more 
rapidly  from  the  3 -hour  forecast  to  the  6-hour  forecast  than  from  the  12-hour  forecast  to 
the  24-hour  forecast.  The  mean  forecast  skill  score,  for  example,  for  the  3 -hour  forecast 
for  all  cases  is  0.40  and  decreases  to  0.30  for  the  6-hour  forecast  and  finally  decreases  to 
0.25  and  0.23  for  the  12  and  24-hour  forecasts  respectively.  This  pattern  is  likewise  seen 
in  the  RMSE  and  is  perhaps  best  illustrated  in  the  MAPD  where  the  differences  for  the  3, 
6,  12  and  24-hour  forecasts  are  21%,  28%,  31%  and  32%  respectively. 
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In  all  four  cases,  GAIM-GM  showed  a  diurnal  forecast  trend.  The  RMSE 
increased  during  daytime  and  decreased  at  night.  Skill  scores  typically  decreased  and  at 
times  became  negative  during  nighttime  as  the  difference  between  the  forecast  and 
reference  RMSE  decreased.  Finally,  with  respect  to  the  diurnal  pattern,  GAIM-GM 
exhibited  a  diurnal  bias  to  under  forecast  at  night  and  over  forecast  during  the  day.  This 
was  seen  in  the  percent  difference  plots  as  the  MPD  reached  minima  at  night  and  maxima 
during  the  day. 

Forecast  RMSE  typically  increased  as  geomagnetic  storming  increased. 
Similarly,  forecast  RMSE  increased  as  solar  activity  increased.  At  times,  as  the  RMSE 
increased,  the  skill  score  increased  as  well,  indicating  that  the  forecast  was  improving 
over  the  reference  in  representing  the  ionosphere. 

There  is  a  positive  correlation  between  average  TEC  values  and  RMSE.  This 
correlation  is  greater  during  geomagnetic  stonning  conditions  than  during  quiet 
geomagnetic  conditions. 

The  GAIM-GM  model  is  able  to  forecast  short  term  weather  phenomena  as  seen 
by  skill  scores  greater  than  zero.  Sunrise,  sunset  and  the  onset  and  ending  of 
geomagnetic  storms  generate  low  and  negative  skill  scores  showing  that  at  times  the 
reference  background  produces  a  more  reliable  forecast  than  the  forecast  output  itself. 
Geomagnetic  storms  and  diurnal  variation  in  the  ionosphere  impact  forecast  accuracy. 
Finally,  solar  minimum  accuracy  during  quiet  conditions  is  greater  than  solar  max 
accuracy. 
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5.2  Recommendations  for  Future  Work 


Owing  to  the  myriad  of  variables  in  the  ionosphere  and  likewise  in  the 
configuration  of  the  model,  it  is  impossible  for  any  one  study  to  fully  determine  model 
forecast  accuracy.  The  work  here  presented  determined  forecast  accuracy  for  GAIM-GM 
on  a  global  scale  with  concentration  forecast  accuracy  over  CONUS. 

To  further  confirm  the  efficiency  of  the  Kalman  filter  and  the  data  assimilation 
process  to  obtain  accurate  forecasts,  this  analysis  should  be  repeated  over  other  data  rich 
regions  such  as  Europe  and  Australia.  This  would  give  further  insight  into  the  diurnal 
error  trends  and  bias. 

This  analysis  investigated  accuracy  with  respect  to  TEC.  GAIM-GM  also 
forecasts  for  the  ionospherically  important  height  (hmF2)  and  density  (NmFi)  of  the  F2 
peak  in  the  ionosphere.  Hence,  the  hmF2  and  NmF2  output  forecasts  accuracy  should  also 
be  investigated  to  determine  overall  accuracy  as  well  as  the  extent  of  diurnal  error  and 
bias  for  these  parameters. 

GAIM-GM  is  a  data  assimilation  model.  The  model  accuracy  was  tested  in  line 
with  how  AFWA  runs  the  model.  The  contributions  by  various  data  platforms  to  forecast 
accuracy  should  be  investigated.  It  is  possible  that  other  types  and  combinations  of  data 
ingests,  as  well  as  data  ingest  locations  could  have  a  significant  impact  on  model  forecast 
accuracy.  Testing  this  may  produce  output  with  greater  accuracy,  determine  potential 
means  to  reduce  or  acquire  data  to  keep  and  improve  model  accuracy.  Work  on  this  topic 
could  impact  budgetary  issues  as  access  to  data  costs  money.  If  acceptable  accuracy  can 
be  achieved  with  less  data,  money  may  be  saved. 


53 


Bibliography 


Delorey,  Dennis  E.,  Pruneau,  Paul  N.,  Parsons,  Carolyn  M.  (1989), 

Database  Development  for  the  DMSP  SSIES  Experiment,  Space  Data  Analysis 
Laboratory,  Chestnut  Hill,  Massachusetts. 

Howe,  B.  M.,  and  K.  Runciman  (1998),  Tomography  of  the  ionosphere:  Four¬ 
dimensional  simulations,  Radio  Sci.,  33(1),  109-128,  doi:  January  2008. 

Maybeck,  P.  S.  (1979),  Stochastic  models,  estimation,  and  control,  Academic  Press,  New 
York. 

Ondoh,  T.,  and  K.  Marubashi  (2001),  Science  of  Space  Environment,  Ohmsha,  Ltd., 
Tokyo,  Japan. 

Scherliess,  L.,  R.  W.  Schunk,  J.  J.  Sojka,  and  D.  C.  Thompson  (2004),  Development  of  a 
physics-based  reduced  state  Kalman  filter  for  the  ionosphere,  Radio  Sci.,  39(1), 
RS1S04. 

Scherliess,  L.,  R.  W.  Schunk,  J.  J.  Sojka,  D.  C.  Thompson,  and  L.  Zhu  (2006),  Utah  State 
University  Global  Assimilation  of  Ionospheric  Measurements  Gauss-Markov 
Kalman  filter  model  of  the  ionosphere:  Model  description  and  validation, 
J.Geophys.Res.,  111{ All),  A 1 1315. 

Schunk,  R.  W.,  and  A.  Nagy  (2009),  Ionospheres:  Physics,  Plasma  Physics,  and 
Chemistry,  Second  ed.,  Cambridge  University  Press,  Cambridge,  United 
Kingdom. 

Schunk,  R.  W.,  and  J.  J.  Sojka  (1995),  Development  of  a  Global  Ionospheric  Forecast 
Model,  Hanscom  AFB,  Ma. 

Schunk,  R.  W.,  et  al  (2004),  Global  Assimilation  of  Ionospheric  Measurements  (GAIM- 
GM),  Radio  Sci.,  39(  1),  RS 1 S02. 

Sojka,  J.  J.,  D.  C.  Thompson,  L.  Scherliess,  R.  W.  Schunk,  and  T.  J.  Harris  (2007), 
Assessing  models  for  ionospheric  weather  specifications  over  Australia  during  the 
2004  Climate  and  Weather  of  the  Sun-Earth-System  (CAWSES)  campaign, 
J.Geophys.Res.,  112(A9),  A09306. 

Thompson,  D.  C.,  L.  Scherliess,  J.  J.  Sojka,  and  R.  W.  Schunk  (2006),  The  Utah  State 
University  Gauss-Markov  Kalman  filter  of  the  ionosphere:  The  effect  of  slant 
TEC  and  electron  density  profile  data  on  model  fidelity,  Journal  of  Atmospheric 
and  Solar-Terrestrial  Physics,  68,  947-958. 

Wilks,  D.  S.  (2006),  Statistical  Methods  in  the  Atmospheric  Sciences,  Volume  100, 
Second  Edition  (International  Geophysics),  Second  ed.,  611  pp.,  Academic  Press, 
Burlington,  Mass. 


54 


Zhu,  L.,  R.  W.  Schunk,  G.  Jee,  L.  Scherliess,  J.  J.  Sojka,  and  D.  C.  Thompson  (2006), 
Validation  study  of  the  Ionosphere  Forecast  Model  using  the  TOPEX  total 
electron  content  measurements,  Radio  Sci.,  41(5),  RS5S1 1. 


55 


Vita 


Lieutenant  Paul  H.  Domin  graduated  from  Brighton  High  School  in  Rochester, 
New  York  in  2000.  After  serving  a  2  year  mission  for  his  church,  Lt  Domin  enrolled  in 
The  State  University  of  New  York  College  at  Brockport  and  graduated  in  2007  with  a 
Bachelor  of  Science  degree  in  Meteorology.  Lt  Domin  was  accepted  to  Officer  Training 
School  at  Maxwell  AFB,  Alabama  where  he  commissioned  as  a  second  Lieutenant  in  the 
United  States  Air  Force. 

His  first  assignment  took  him  overseas  to  the  21st  Operational  Weather  Squadron 
at  Sembach  AB,  Germany.  There  he  fulfilled  the  roles  of  Lead  Meteorologist  and 
Assistant  Flight  Commander.  After  less  than  two  years  in  Germany,  Lt  Domin  was 
selected  for  enrollment  into  the  Space  Physics  program  at  the  Air  Force  Institute  of 
Technology  at  Wright-Patterson  AFB,  Ohio.  Upon  graduation,  he  will  serve  at  the  Air 
Force  Weather  Agency,  Offutt  AFB,  Nebraska. 


56 


REPORT  DOCUMENTATION  PAGE 


Form  Approved 
OMB  No.  074-0188 


The  public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions, 
searching  existing  data  sources,  gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments 
regarding  this  burden  estimate  or  any  other  aspect  of  the  collection  of  information,  including  suggestions  for  reducing  this  burden  to  Department  of  Defense, 
Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports  (0704-0188),  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington, 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  an  penalty  for  failing  to  comply 
with  a  collection  of  information  if  it  does  not  display  a  currently  valid  OMB  control  number. 

PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 


1.  REPORT  DATE  (DD-MM-YYYY) 

16-09-2011 


TITLE  AND  SUBTITLE 


2.  REPORT  TYPE 

Master’s  Thesis 


3.  DATES  COVERED  (From  -  To) 
January  2010  -  September  2011 


5a.  CONTRACT  NUMBER 


Verification  of  Global  Assimilation  of  Ionospheric  Measurements  Gauss  Markov 
(GAIM-GM)  Model  Forecast  Accuracy 


5b.  GRANT  NUMBER 


5c.  PROGRAM  ELEMENT  NUMBER 


6.  AUTHOR(S) 

Domm,  Paul  H,  1st  Lieutenant,  USAF 


5d.  PROJECT  NUMBER 


5e.  TASK  NUMBER 


5f.  WORK  UNIT  NUMBER 


7.  PERFORMING  ORGANIZATION  NAMES(S)  AND  ADDRESS(S) 

Air  Force  Institute  of  Technology 

Graduate  School  of  Engineering  and  Management  (AFIT/ENY) 
2950  Hobson  Way,  Building  640 
WPAFB  OH  45433-8865 


9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

Air  Force  Weather  Agency 

101  Nelson  Drive 

Offutt  AFB,  NE  68113 

DSN  271-0690,  COMM  402-294-0690 

Email:  2syosdor@offutt.af.mil 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  UNLIMITED. 


8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 

AFIT/GAP/ENP/1 1-S01 


10.  SPONSOR/MONITOR’S 
ACRONYM(S) 

AFWA 


11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 


13.  SUPPLEMENTARY  NOTES 


14.  ABSTRACT 


GAIM-GM  is  an  operational  Kalman  filter  data  assimilation  model  of  the  ionosphere  that  can  assimilate 
data  from  GPS  total  electron  content  (TEC),  ionosonde  electron  density  profiles,  and  satellite  based  in  situ  electron 
densities.  The  Air  Force  Weather  Agency  (AFWA)  uses  GAIM-GM  to  specify  and  forecast  the  ionosphere.  An  in 
depth  investigation  into  the  accuracy  of  these  forecasts  has  not  been  completed.  GAIM-GM  output  obtained  from 
four  cases  run  from  combinations  of  geomagnetic  and  solar  activity  was  used  to  determine  GAIM-GM  forecast 
accuracy.  Forecast  accuracy  was  determined  through  the  use  of  a  skill  score  as  well  as  other  statistical  tools  to 
include  root  mean  square  error  (RMSE)  and  percent  differences  between  the  forecasts  and  the  observations.  It  is 
determined  that  overall,  GAIM-GM  forecast  output  is  more  representative  than  the  climatological  reference  90%  of 
the  time  indicating  that  GAIM-GM  can  therefore  forecast  short  term  fluctuations  in  the  ionosphere.  The  RMSE 
increases  during  daytime,  geomagnetic  storming  and  during  solar  maximum.  Skill  scores  decrease  at  night  and 
GAIM-GM  shows  an  overall  diurnal  bias  to  under  forecast  at  night  and  over  forecast  during  the  day. 


15.  SUBJECT  TERMS 

Ionosphere,  Ionospheric  Forecast  Model,  Global  Assimilation  of  Ionospheric  Measurements  Gauss 
Markov  Kalman  Filter  (GAIM-GMKF),  Space  Weather 


16.  SECURITY  CLASSIFICATION 
OF: 


17.  LIMITATION 

18. 

19a.  NAME  OF  RESPONSIBLE  PERSON 

OF 

ABSTRACT 

NUMBER 

Robb  M.  Randall,  Lt  Col,  USAF  ADVISOR 

OF  PAGES 

19b.  TELEPHONE  NUMBER  (Include  area  code) 

uu 

68 

(402)  294-9544,  (robb.randall@us.af.mil) 

57 


